Image processing apparatus, image processing method, and image capturing apparatus

ABSTRACT

Face regions are detected from a captured image, and a weight of each detected face region is computed based on a size and/or a position of the detected face region. Then a previous priority ranking weight is computed based on a priority ranking determined in previous processing. A priority of the face region is computed from the weight and the previous priority ranking weight. For example, if the continuous processing number exceeds the threshold the priority ranking weight is reduced. After the processing is completed for all face regions, a priority ranking of each face region is determined based on the priority computed for each face region.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method, and an image capturing apparatus, and in particularrelates to an image processing apparatus, an image processing method,and an image capturing apparatus that can detect a specific subject suchas a person, an animal, or an object included in an image, or a part ofthe subject.

2. Description of the Related Art

An image processing technique of automatically detecting a specificsubject from an image is highly useful, and can be utilized, forexample, for specifying a face region of a person in a moving image.Such an image processing technique can be employed in many fieldsincluding teleconferences, man-machine interfaces, security systems,monitor systems for tracking human faces, and image compression. Adigital camera or digital video camera that detects a face from acaptured image and optimizes focus or exposure for the detected face isalso known. Japanese Patent Laid-Open No. 2005-318554 discloses anapparatus that focuses on a face detected from an image and captures animage at an optimum exposure for the face.

Furthermore, there is a known image capturing apparatus that has afunction of selecting a subject that serves as a main subject from aplurality of subjects, in order to perform more accurate AF (Autofocus)and the like even in the case where a plurality of specific subjects arepresent in an image.

Japanese Patent Laid-Open No. 2002-51255 discloses an image capturingapparatus that selects a subject that serves as a main subject based onthe states of the subjects present in an image. A state of a subjectrefers to various parameters indicating, for example, a distance fromthe image capturing apparatus to the subject, an area of the subject inthe image, a position of the subject in the image, a distance betweensubjects, and the like. Moreover, Japanese Patent Laid-Open No.2008-5438 discloses an image capturing apparatus that suppressesfrequent switching of a main subject, by selecting a current mainsubject in consideration of information about a previously selected mainsubject.

One method for selecting a main subject from a plurality of subjectsaccording to a conventional technique is a method of performing a mainsubject selection process based only on the states of subjects at aspecific time. This method has the advantage of enabling the appropriatemain subject at the specific time to be selected, but also has theproblem of causing frequent switching of the main subject between aplurality of subjects.

This problem is described below, with reference to FIG. 1. In an exampleshown in FIG. 1, subjects 100 and 101 are each detected by an imagecapturing apparatus in frames #1 to #4 of a moving image inchronological order. In FIG. 1, a main subject is indicated by a solidline box, and a subject other than the main subject is indicated by adashed line box.

As an example, the main subject is determined according to a prioritythat is computed based on a distance from the image center to thegravity center of a subject image and a size of the subject image, inthe states of the detected subjects 100 and 101. Suppose the distancefrom the image center to the subject gravity center and the subjectsize, which indicate the state of the subject, are similar between thetwo subjects 100 and 101, as shown in FIG. 1. In such a case, slightchanges in the states of the two subjects 100 and 101 cause frequentswitching of the main subject between the subjects 100 and 101. That is,priority rankings of the detection results change.

In the case of performing AF and the like on the main subject, when themain subject is frequently switched between a plurality of subjects,such changes of the main subject cause an AF lens and the like to befrequently driven. This induces a possibility of reducing theoperational stability and leading to a loss of accuracy of focusing andthe like. In addition, in the case where a marker or the like indicatingthe main subject is displayed in the display of the image capturingapparatus, such frequent changes of the marker display as shown inframes #1 to #4 in FIG. 1 are annoying to the user.

Another method for selecting a main subject from a plurality of subjectsis a method of performing a main subject selection process based on thestates of subjects at a specific time and information on a previouslyselected main subject. This method has the advantage of suppressingfrequent changes of the main subject, but also has the problem of makingit difficult to switch the main subject even in a scene where the mainsubject needs to be switched.

This problem is described below, with reference to FIG. 2. In an exampleshown in FIG. 2, subjects 102 and 103 are each detected by an imagecapturing apparatus in frames #1 to #4 of a moving image inchronological order. In FIG. 2, a main subject is indicated by a solidline box, and a subject other than the main subject is indicated by adashed line box.

As an example, the main subject is determined according to a prioritythat is computed based on a distance from the image center to thegravity center of a subject image, a size of the subject image, and apriority ranking in a previous frame, in the states of the detectedsubjects 102 and 103. To suppress frequent switching of the mainsubject, it is necessary to increase, in the priority computation, theinfluence of the priority ranking in the previous frame to a certainextent.

In detail, priority rankings of the subjects 102 and 103 in frame #1 inFIG. 2 are determined first. In this example, the subject 102 isdetermined to have a higher priority ranking than the subject 103, sothat the subject 102 is selected as the main subject. Once the mainsubject has been determined, in the following frames #2 to #4 asituation where it is difficult to switch the main subject arises, dueto the influence of the priority rankings in frame #1.

If the main subject is not switched from the subject 102 to the subject103 even when the states of the subjects in the captured image are asshown in frame #4 in FIG. 2, the user may feel something is wrong.

SUMMARY OF THE INVENTION

To solve at least one of these problems of the conventional techniques,the present invention provides an image processing apparatus, an imageprocessing method, and an image capturing apparatus that can determinean appropriate priority ranking for a detected subject.

According to one aspect of the present invention, there is provided animage processing apparatus that determines a priority ranking of asubject detected from an image, the image processing apparatuscomprising: a detection unit that detects one or more subjects from animage; a computation unit that computes a priority for determining apriority ranking, for each of the subjects detected by the detectionunit; and a determination unit that determines the priority ranking ofeach of the subjects detected by the detection unit, based on thepriority computed by the computation unit, wherein the computation unit:assigns a larger weight to a subject detected by the detection unit atleast one of when the subject has a larger size and/or when the subjectis positioned nearer a center of the image; corrects the assigned weightby a larger weight when a most recent priority ranking determined forthe subject by the determination unit in a previous image from which thesubject is detected is higher; computes a higher priority for thesubject when the corrected weight is larger; and, when a predeterminedcondition is satisfied, computes the priority with a reduced amount ofcorrection by the weight of the most recent priority ranking.

According to another aspect of the present invention, there is providedan image processing method for determining a priority ranking of asubject detected from an image, the image processing method comprising:a detection step of detecting one or more subjects from an image; acomputation step of computing a priority for determining a priorityranking, for each of the subjects detected in the detection step; and adetermination step of determining the priority ranking of each of thesubjects detected in the detection step, based on the priority computedin the computation step, wherein the computation step includes:assigning a larger weight to a subject detected in the detection step atleast one of when the subject has a larger size and/or when the subjectis positioned nearer a center of the image; correcting the assignedweight by a larger weight when a most recent priority ranking determinedfor the subject in the determination step in a previous image from whichthe subject is detected is higher; computing a higher priority for thesubject when the corrected weight is larger; and, when a predeterminedcondition is satisfied, computing the priority with a reduced amount ofcorrection by the weight of the most recent priority ranking.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram explaining a problem of a conventional technique.

FIG. 2 is a diagram explaining a problem of a conventional technique.

FIG. 3 is a block diagram showing an example of a structure of an imagecapturing apparatus that is commonly applicable to each embodiment ofthe present invention.

FIG. 4 is a flowchart showing an example of processing for determiningpriority rankings of face regions according to a first embodiment of thepresent invention.

FIG. 5A is a diagram showing an example of a relationship between adistance from the image center to the gravity center of a face regionand a position weight of the face region.

FIG. 5B is a diagram showing an example of a relationship between a sizeof a face region and a size weight of the face region.

FIG. 6 is a diagram showing an example of a relationship between aprevious priority ranking of a face region and a previous priorityranking weight of the face region.

FIG. 7 is a diagram explaining the processing according to the firstembodiment of the present invention in more detail.

FIG. 8 is a flowchart showing an example of processing for determiningpriority rankings of face regions according to a second embodiment ofthe present invention.

FIG. 9 is a diagram explaining the processing according to the secondembodiment of the present invention in more detail.

FIG. 10 is a flowchart showing an example of processing for determiningpriority rankings of face regions according to a third embodiment of thepresent invention.

FIG. 11 is a diagram explaining the processing according to the thirdembodiment of the present invention in more detail.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail in accordance with the accompanying drawings.

FIG. 3 shows an example of a structure of an image capturing apparatus300 that is commonly applicable to each embodiment of the presentinvention. An image capture unit 302 includes an imaging optical system,an image sensor such as a CCD image sensor or a CMOS image sensor, and adrive circuit that drives the image sensor. Light from a subject iscollected by a photographic lens 301 provided in the imaging opticalsystem and applied to the image sensor in the image capture unit 302. Inthe image capture unit 302, the incident light is converted to a chargeper pixel by photoelectric conversion in the image sensor. The imagecapture unit 302 reads the charge converted from the light per pixel,and outputs the read charge as an image signal.

Note that a moving image signal can be obtained by sequentially readingcharges from the image sensor at a predetermined time interval such as aframe period in the image capture unit 302.

The image signal output from the image capture unit 302 undergoes analogsignal processing such as correlated double sampling (CDS) and gainadjustment in an analog signal processing unit 303. The image signaloutput from the analog signal processing unit 303 is converted to adigital signal in an A/D conversion unit 304, as image data.

The image data output from the A/D conversion unit 304 is supplied toeach of an image capture control unit 305 and an image processing unit306. The image processing unit 306 performs predetermined imageprocessing, such as gamma correction and white balancing, on thesupplied image data. In addition to ordinary image processing, the imageprocessing unit 306 has a function of performing image processing thatuses information relating to a region of a face in an image, which issupplied from a face detection unit 309 described later. The imageprocessing unit 306 can also perform a process of correcting imageblurring caused by hand movement, based on angular velocity informationoutput from a gyro sensor (not shown).

The image data output from the image processing unit 306 is supplied toa display unit 307. The display unit 307 as a display means includes adisplay device such as an LCD or an organic EL display, and a drivecircuit that drives the display device based on the image data. Thedisplay unit 307 displays the supplied image data on the display device.By sequentially displaying images that are serially captured inchronological order in the display unit 307, the display unit 307 canfunction as an electronic view finder (EVF) for monitoring capturedimages.

The image data output from the image processing unit 306 is alsosupplied to a recording unit 308 and recorded on a recording medium 312.As an example, the recording medium 312 is a nonvolatile semiconductormemory that can be removably attached to the image capturing apparatus300. The present invention is not limited to this and the recordingmedium 312 may be an internal memory or internal hard disk (not shown)included in the image capturing apparatus 300, or an external apparatuscommunicably connected via a communication unit (not shown). Therecording medium 312 may also be an optical disc or magnetic tape.

The image data output from the image processing unit 306 is alsosupplied to the face detection unit 309. The face detection unit 309detects a human face in an image, and specifies the number of personsthat are subjects and the face regions of those persons. The facedetection unit 309 can perform face detection using a known facedetection method.

Known face detection techniques include a method of using face-relatedknowledge (skin tone information, parts such as the eyes, nose, andmouth), a method of configuring a discriminator for face detectionaccording to a learning algorithm represented by a neural network, andso on. These methods are typically combined to perform face recognition,in order to improve a face recognition rate. As a concrete example, aface detection method using a wavelet transform and an image featurequantity is described in Japanese Patent Laid-Open No. 2002-251380.

The face detection unit 309 outputs, for example, a position, a size, atilt, a degree of reliability, and the like for each of the faces of alldetected persons, as face region information. The degree of reliabilityreferred to here is a value indicating how reliable the detection resultis, and is determined during the face detection process.

One example of a method for computing the degree of reliability is amethod of comparing a feature of an image of a subject stored beforehandand a feature of an image of a face region detected by the facedetection unit 309 to obtain a probability that the image of thedetected face region is the image of the subject, and computing thedegree of reliability from this probability. For instance, a face imageaccording to a typical face or an image obtained by extracting andgraphically representing a facial feature may be used as the image ofthe subject stored beforehand. Another example of the method forcomputing the degree of reliability is a method of computing adifference between a feature of an image of a subject stored beforehandand a feature of an image of a face region detected by the facedetection unit 309 and computing the degree of reliability from thedifference. The degree of reliability of the face detection may becomputed using other methods. When the output degree of reliability ishigh, the probability of false detection is low. When the output degreeof reliability is low, the probability of false detection is high.

The face region information detected by the face detection unit 309 issupplied to a priority determination unit 310. First, the prioritydetermination unit 310 determines detection results each of which can beregarded as a face as intended, namely, face regions, from informationsuch as the degree of reliability included in the detected face regioninformation. As an example, it is assumed that a face region whosedegree of reliability is not lower than a threshold can be regarded as aface as intended.

The priority determination unit 310 then computes a priority of eachdetection result (hereafter referred to as “face region”) determined tobe regarded as a face as intended, based on a state of a face and aprevious priority ranking. The state of the face includes a distancefrom the image center to the gravity center of the face image, and asize of the face image. The extent to which the previous priorityranking is taken into consideration in the priority computation changesdepending on a predetermined condition described later. Note thatinformation on the previous priority ranking is supplied to the prioritydetermination unit 310 from a priority ranking determination unit 311described later. The priority determination unit 310 supplies a resultof the computing of the priority of each face region, to the priorityranking determination unit 311.

The priority ranking determination unit 311 determines a priorityranking of each face region, according to the priority of each faceregion computed by the priority determination unit 310. A face having ahigher priority is given a higher priority ranking. Each face regionprioritized by the priority ranking determination unit 311 is suppliedto the image processing unit 306 and the image capture control unit 305.Each face region prioritized by the priority ranking determination unit311 is also supplied to the priority determination unit 310 as aprevious priority ranking result used in priority computation. Thepriority determination unit 310 uses this priority ranking resultsupplied from the priority ranking determination unit 311, as theprevious priority ranking when computing the priority of each faceregion.

The image capture control unit 305 controls a focusing control mechanismand an exposure control mechanism (not shown) in the photographic lens301, based on the image data output from the A/D conversion unit 304.When controlling the focusing control mechanism and the exposure controlmechanism, the image capture control unit 305 can use the prioritizedface region information supplied from the priority ranking determinationunit 311. For example, the focusing control mechanism sets only a faceof a face region having the highest priority ranking of the facesdetected by the face detection unit 309, as a control target. Forexample, the exposure control mechanism performs processing on, as acontrol target, faces of face regions having the highest to thirdhighest rankings of the faces detected by the face detection unit 309.

Thus, the image capturing apparatus 300 in this embodiment has afunction of performing an imaging process in consideration of theinformation on face regions in a captured image. The image capturecontrol unit 305 also controls output timings, output pixels, and thelike of the image sensor. The image capture control unit 305 furtherperforms a process of driving a zooming mechanism (not shown) providedin the imaging optical system, according to a user operation.

A control unit 320 controls an entire operation of the image capturingapparatus 300. For example, the control unit 320 includes a CPU, a ROM,and a RAM, and controls each unit of the image capturing apparatus 300by operating through the use of the RAM as a work memory according to aprogram stored in the ROM beforehand. Here, the functions of thepriority determination unit 310 and the priority ranking determinationunit 311 described above can be realized by the program in the controlunit 320. Furthermore, the function of the face detection unit 309 maybe realized by the program. Alternatively, these functions may berealized by hardware controlled by the control unit 320.

First Embodiment

The following describes a first embodiment of the present invention.FIG. 4 is a flowchart showing an example of processing for determiningpriority rankings of face regions according to the first embodiment. Theprocesses in this flowchart are executed by the control unit 320controlling the face detection unit 309, the priority determination unit310, and the priority ranking determination unit 311 according to theprogram. The present invention is not limited to this and the processesin the flowchart of FIG. 4 may be executed autonomously by the facedetection unit 309, the priority determination unit 310, and thepriority ranking determination unit 311 in cooperation with each other.

First, in step S401, the control unit 320 initializes a continuousprocessing number for counting the number of times the series ofprocesses is performed to 0. In step S402 which follows, the facedetection unit 309 performs a face detection process on the image dataof one frame, thereby detecting face regions in the image. The number offace regions in the image, a position, a size, and a degree ofreliability of each face region in the image, and the like are obtainedas information on the detected face regions. Here, the control unit 320sets, based on the face detection results by the face detection unit309, only the results that are determined to be faces as valid data. Thecontrol unit 320 then advances the processing to step S403.

In step S403, the control unit 320 determines whether or not prioritycomputation by the processes of steps S404 to S409 described later hasbeen performed for every result determined to be valid as a face in stepS402, that is, every face region detected in the image. When the controlunit 320 determines that priority computation has been performed forevery valid face region, the control unit 320 advances the processing tostep S410. The process in step S410 will be described later. On theother hand, when the control unit 320 determines in step S403 thatpriority computation has not been performed for every valid face region,the control unit 320 advances the processing to step S404.

The processes in steps S404 to S409 are performed for each face regiondetected in the image. In step S404, the priority determination unit 310determines a position weight of the face region based on a distance fromthe image center to the gravity center of the face region. FIG. 5A showsan example of a relationship between the distance from the image centerto the gravity center of the face region and the position weight of theface region. In FIG. 5A, a vertical axis represents the position weight,and a horizontal axis represents the distance. For example, when thegravity center of the face region is positioned at the image center, theposition weight is 1.0. As the distance from the image center to thegravity center of the face region increases, the position weight isdecreased. When the distance from the image center to the gravity centerof the face region is not smaller than a predetermined distance, theposition weight is fixed to a certain value such as 0.2. That is, theposition weight is a fixed value near the image edges.

Though the position weight is reduced in a linear fashion in the rangefrom the image center to the predetermined distance in FIG. 5A, thisexample is not a limit on the present invention. The position weight mayinstead be reduced in a curved fashion such as an exponential functionor a logarithmic function, with respect to the distance. Moreover, it isassumed that the relationship between the distance from the image centerto the gravity center of the face region and the position weight of theface region as exemplified in FIG. 5A is held in the ROM or the like asa table by the control unit 320 or the priority determination unit 310beforehand. The present invention is not limited to this and therelationship between the distance from the image center to the gravitycenter of the face region and the position weight of the face region maybe computed using a computational expression in the program.

In step S405 which follows, the priority determination unit 310determines a size weight of the face region based on a size of the faceregion. FIG. 5B shows an example of a relationship between the size ofthe face region and the size weight of the face region. In FIG. 5B, avertical axis represents the size weight, and a horizontal axisrepresents the size of the face region. For example, when the size ofthe face region is not smaller than a predetermined value, the sizeweight is 1.0. As the size of the face region becomes smaller, the sizeweight is reduced. When the size of the face region is not larger thananother predetermined value such as a value close to a minimum sizedetectable by the face detection unit 309, the size weight is fixed to acertain value such as 0.2.

Though the size weight is reduced in a linear fashion in the range wherethe size of the face region is from the threshold to the detectableminimum size in FIG. 5B, this example is not a limit on the presentinvention. The size weight may instead be reduced in a curved fashionsuch as an exponential function or a logarithmic function, with respectto the size. Moreover, it is assumed that the relationship between thesize of the face region and the size weight of the face region asexemplified in FIG. 5B is held in the ROM or the like as a table by thecontrol unit 320 or the priority determination unit 310 beforehand. Thepresent invention is not limited to this and the relationship betweenthe size of the face region and the size weight of the face region maybe computed using a computational expression in the program.

After the size weight of the face region is determined in step S405, thecontrol unit 320 advances the processing to step S406. Note that onlyone of the processes of steps S404 and S405 may be performed. In stepS406, the control unit 320 determines whether or not the continuousprocessing number exceeds a threshold. When the control unit 320determines that the continuous processing number exceeds the threshold,the control unit 320 resets the continuous processing number to 0 instep S408, and then advances the processing to step S409. The process ofstep S409 will be described later.

When the control unit 320 determines that the continuous processingnumber is not more than the threshold, on the other hand, in step S407the priority determination unit 310 obtains a previous priority ranking,that is, the most recent priority ranking, determined for the faceregion that is the target of processing, from the priority rankingdetermination unit 311. The priority determination unit 310 determines aprevious priority ranking weight of the face region based on theobtained previous priority ranking.

FIG. 6 shows an example of a relationship between the previous priorityranking of the face region and the previous priority ranking weight ofthe face region. In FIG. 6, a vertical axis represents the previouspriority ranking weight, and a horizontal axis represents the previouspriority ranking. When the previous priority ranking is higher, theprevious priority ranking weight is higher. As the previous priorityranking decreases, the previous priority ranking weight is reduced. Whenthe previous priority ranking is not higher than a predeterminedranking, the previous priority ranking weight is fixed to apredetermined minimum value. In the example of FIG. 6, when the previouspriority ranking result is highest (first), the previous priorityranking weight is 1.4. When the previous priority ranking result issecond or third, the previous priority ranking weight is 1.2. When theprevious priority ranking is not higher than the predetermined ranking(not higher than fourth in the example of FIG. 6) or when the faceregion has not been detected as valid data in the previous processingresult, the previous priority ranking weight is 1.0.

Here, to reflect the previous priority ranking result on the currentresult, it is necessary to establish a correspondence relationship forthe face region between the currently processed frame and the previouslyprocessed frame. For example, the priority determination unit 310compares the position and the size of the face region according to thecurrent detection result with a position and a size of a face regionaccording to the previous detection result. When they have at least apredetermined level of similarity as a result of comparison, thepriority determination unit 310 determines that they are the samesubject. In this way, the previous priority ranking result can bereflected on the current result.

As described above, when the continuous processing number exceeds thethreshold in step S406, the continuous processing number is reset instep S408 without performing the process in step S407. Thus, theinfluence of the previous priority ranking weight is cancelled once in apredetermined number of times (not less than 2).

After determining the previous priority ranking weight in step S407, thepriority determination unit 310 computes, in step S409, a priority ofthe face region using the following expression (1) based on the positionweight, the size weight, and the previous priority ranking weightdetermined respectively in steps S404, S405, and S407 described above.

(Priority)=(Position weight)×(Size weight)×(Previous priority rankingweight)  (1)

In the case where the continuous processing number is determined toexceed the threshold in step S406 and so the processing advances to stepS409 without performing the process of step S407, the prioritydetermination unit 310 uses, for example, 1 as the previous priorityranking weight. Likewise, in the case of processing the first frame,with there being no previous processing, the priority determination unit310 uses, for example, 1 as the previous priority ranking weight.

After the priority determination unit 310 computes the priority of aface region in step S409, the processing returns to step S403. Asmentioned earlier, when the control unit 320 determines in step S403that the processes of steps S404 to S409 have been performed for everyvalid face region detected in the image, the processing advances to stepS410.

In step S410, the priority ranking determination unit 311 determines apriority ranking for each of the face regions detected in the image,based on the priority computed in step S409. Here, the face regions aregiven the decreasing priority rankings in decreasing order of priority.The determined priority ranking of each face region is held in thepriority ranking determination unit 311, and also associated with faceregion information (such as the position, the size, and the like of eachface region in the image) and supplied to the image capture control unit305. The image capture control unit 305 performs focusing control andexposure control as mentioned above, based on the supplied priorityranking of each face region.

After the priority ranking determination unit 311 determines thepriority ranking of each face region in step S410, the processingadvances to step S411 where the control unit 320 increments thecontinuous processing number, and then the processing returns to stepS402. Note that the loop processing of steps S402 to S411 is repeated ata predetermined rate, such as a period of one frame or a period of aplurality of frames. The present invention is not limited to this andthe loop processing may be repeated in accordance with a time periodrequired for the face detection process by the face detection unit 309.

The processing according to the flowchart of FIG. 4 mentioned above isdescribed in more detail below, with reference to FIG. 7. In an exampleshown in FIG. 7, face regions 200 to 203 that are substantially equal insize are each detected in frames #1, #2, . . . , #n, #n+1 of a movingimage in chronological order.

In FIG. 7, a face region having the highest priority ranking isindicated by a solid line box, and face regions having the second andthird highest priority rankings are indicated by a dashed line box. Aface region having the fourth highest priority ranking or below is shownwithout a box. For instance, the image capture control unit 305 uses theface region having the highest priority ranking, as the control targetof focusing control and exposure control. Moreover, the control unit 320adds box displays to the face regions having the highest to thirdhighest priority rankings, upon the display by the display unit 307.Such box displays indicate subjects whose priority rankings are notlower than a predetermined ranking.

In FIG. 7, in the first frame #1, there is no previous priority rankingdetermination result, so that the priority rankings are determinedaccording to the position weights and the size weights. In the exampleof FIG. 7, the face region 202 positioned at the image center is giventhe highest priority ranking. Meanwhile, the face region 200 is farthestfrom the image center among the face regions 200 to 203, and so is giventhe lowest priority ranking.

In the next frame #2, the priority rankings are determined inconsideration of the previous priority rankings, for face regions thatare found to correspond to the face regions detected in frame #1. In thecase where the threshold used in step S406 in FIG. 4 is n, the priorityrankings are determined in consideration of the previous priorityranking result up to frame #n in FIG. 7. This being so, when there is nosubstantial change in the states of the detected face regions, a changeof the priority rankings is suppressed. For example, even when there isa substantial change of state from frame #1 as in the case of the faceregions 200 and 201 in frame #n, the priority rankings are maintained.

On the other hand, in frame #n+1, the process of step S407 is skippedsince the continuous processing number exceeds the threshold, so thatthe previous priority ranking result is not taken into consideration.Therefore, the priority ranking of each face region is determinedaccording to the position weight and the size weight. In other words,the priority ranking is determined according to the current states ofthe face regions in the image, as a result of which the state changes ofthe subjects are reflected more on the priority ranking of each faceregion. In the example of FIG. 7, the face region 201 farthest from theimage center is given the lowest priority ranking.

The first embodiment describes the case where, each time thepredetermined continuous processing number is reached, the priorityranking of each face region is determined without taking the previouspriority ranking into consideration. However, this example is not alimit on the present invention. For instance, the priority ranking maybe determined by changing how much the previous priority ranking istaken into consideration, each time the predetermined continuousprocessing number is reached.

According to the first embodiment described above, frequent switching ofthe priority ranking of each face region is suppressed during the timeperiod when the previous priority ranking is taken into consideration.Moreover, by providing a timing in which the previous priority rankingis not taken into consideration or changing how much the previouspriority ranking is taken into consideration, a priority ranking thatreflects the state of the subject at a specific time is determined.Hence, appropriate priority rankings in consideration of the statechanges of the subjects with time can be determined while suppressingfrequent switching in the priority rankings.

Furthermore, since the priority ranking of each subject can beappropriately determined, it is possible to not only determine the mainsubject, but also determine the priority ranking suitable for eachpurpose, such as AF, AE (automatic exposure), and display, among whichthe priority subject may different.

Second Embodiment

The following describes a second embodiment of the present invention.When compared with the processing in the first embodiment describedabove, the second embodiment differs in the timing in which the previouspriority ranking result is not taken into consideration in the prioritydetermination unit 310. That is, in the second embodiment, the previouspriority ranking result is not taken into consideration in the timingwhen an amount of change between the currently processed image and theimage used when determining the previous priority ranking is found to benot smaller than a predetermined amount. In the second embodiment,whether or not a face region whose previous priority ranking is highestis present in the current image is detected as this amount of change.

FIG. 8 is a flowchart showing an example of processing for determiningpriority rankings of face regions according to the second embodiment.For example, the processes in this flowchart are executed by the controlunit 320 controlling the face detection unit 309, the prioritydetermination unit 310, and the priority ranking determination unit 311according to a program, as with the flowchart of FIG. 4 described above.

The processes of steps S901 to S904 are the same as the processes ofsteps S402 to S405 in FIG. 4 described in the first embodiment. Indetail, in step S901, the face detection unit 309 performs a facedetection process on the image data of one frame to detect face regionsin the image, and obtains information on the detected face regions. Instep S902, which follows, the control unit 320 determines whether or nota priority has been computed for every face region detected in theimage. When the control unit 320 determines that a priority has beencomputed for every detected face region, the processing advances to stepS908. In step S908, the priority ranking determination unit 311determines a priority ranking of each face region detected in the imagebased on the computed priority, as in step S410 in FIG. 4.

When the control unit 320 determines in step S902 that a priority hasnot been computed for every detected face region, on the other hand, theprocessing advances to step S903 where the priority determination unit310 computes a position weight of a face region as in step S404 in FIG.4. In step S904, which follows, the priority determination unit 310computes a size weight of the face region as in step S405 in FIG. 4.

The processing then advances to step S905, and the prioritydetermination unit 310 determines whether or not a face region whoseprevious priority ranking is highest is included in the face regionspresent in the current image. For example, the priority determinationunit 310 obtains information on the face region whose previous priorityranking is highest from the priority ranking determination unit 311. Thepriority determination unit 310 then determines whether or not a faceregion corresponding to the obtained face region is present in thecurrent image. When the priority determination unit 310 determines thatthe corresponding face region is not present, the processing advances tostep S907. The process of step S907 will be described later.

When the priority determination unit 310 determines in step S905 thatthe face region whose previous priority ranking is highest is present inthe current image, on the other hand, the processing advances to stepS906. In step S906, the priority determination unit 310 obtains a weightof a previous priority ranking, that is, the most recent priorityranking, for the face region as in step S407 in FIG. 4.

Here, to determine whether or not the face region whose previouspriority ranking is highest is present in the image, it is necessary toestablish a correspondence relationship in the face region between thecurrently processed frame and the previously processed frame. As anexample, the priority determination unit 310 compares a position and asize of the face region according to the previous detection result, witha position and a size of a face region according to the currentdetection result. When they have at least a predetermined level ofsimilarity as a result of comparison, the priority determination unit310 determines that they are the same subject. This makes it possible todetermine whether or not the face region whose previous priority rankingis highest is present in the current image. Hence, the previous priorityranking result can be reflected on the face region present in thecurrent image.

In step S907, which follows, the priority determination unit 310computes a priority of the face region as in step S409 in FIG. 4. Indetail, in step S907, the priority determination unit 310 computes thepriority of the face region using the above expression (1), based on theposition weight, the size weight, and the previous priority rankingweight determined respectively in steps S903, S904, and S906 describedabove.

After the priority of the face region is computed, the processingreturns to step S902. As mentioned earlier, when the control unit 320determines in step S902 that a priority has been computed for everydetected face region, the processing advances to step S908. In stepS908, the priority ranking determination unit 311 determines thepriority ranking of each face region detected in the image in decreasingorder of priority based on the priority computed in step S907, as instep S410 in FIG. 4. The determined priority ranking is held in thepriority ranking determination unit 311, and also associated with faceregion information and supplied to the image capture control unit 305.

After the priority ranking of each face region is determined in stepS908, the processing returns to step S901. Note that the loop processingof steps S901 to S908 is repeated at a predetermined rate, such as aperiod of one frame or a period of a plurality of frames, or accordingto a time period required for the face detection by the face detectionunit 309.

The processing according to the flowchart of FIG. 8 mentioned above isdescribed in more detail below, with reference to FIG. 9. In an exampleshown in FIG. 9, face regions 210 to 213 that are substantially equal insize are each detected in frames #1 to #4 of a moving image, with anobject 214 being present in front of a subject of the face region 212.

In FIG. 9, in the first frame #1, there is no previous priority rankingdetermination result, so that the priority rankings are determinedaccording to the position weights and the size weights. In the exampleof FIG. 9, the face region 212 positioned near the image center is giventhe highest priority ranking. Meanwhile, the face region 210 is farthestfrom the image center among the face regions 210 to 213, and so is giventhe lowest priority ranking.

In the next frame #2, the priority rankings are determined inconsideration of the previous priority rankings, for face regions thatare found to correspond to the face regions detected in frame #1. Thisbeing so, when there is no substantial change in the states of the faceregions in the image, a change in the priority rankings is suppressed.

In frame #3, the face region 212 whose previous priority ranking, thatis, priority ranking in frame #2, is highest is hidden behind the object214 and is not detected by the face detection unit 309. In this case, byskipping step S906 as a result of the determination process of stepS905, the priority of each face region is computed without taking theprevious priority ranking result into consideration. Accordingly, thepriority rankings of the face regions are determined according to thecurrent states of the face detection results, as a result of which thestate changes of the subjects are reflected more on the priorityrankings.

In the next frame #4, the face region whose previous priority ranking ishighest is present in the current image, so that the priority rankingsare determined in consideration of the previous priority rankings.

The second embodiment describes the case where the priority ranking ofeach face region is determined without taking the previous priorityranking into consideration when the face region whose previous priorityranking is highest is not present. However, this example is not a limiton the present invention. For instance, the priority ranking may bedetermined by changing how much the previous priority ranking is takeninto consideration, depending on the presence of the face region havingthe highest previous priority ranking.

Moreover, the priority ranking may be determined without taking theprevious priority ranking into consideration when the ranking of theface region whose previous priority ranking is highest changes. Anexample of this is given below. First, the priority ranking isdetermined in the current frame while taking the previous priorityranking into consideration. As a result, when the face region whoseprevious priority ranking is highest is given the second highestpriority ranking or below, the priority ranking is recomputed withouttaking the previous priority ranking into consideration.

According to the second embodiment described above, frequent switchingof the priority ranking of each face region is suppressed during thetime period when the previous priority ranking is taken intoconsideration. Moreover, by providing a timing in which the previouspriority ranking is not taken into consideration or how much theprevious priority ranking is taken into consideration is changed, apriority ranking that reflects the state of the subject at a specifictime is determined. That is, the previous priority ranking is not takeninto consideration or how much the previous priority ranking is takeninto consideration is changed, in the case where the face region whoseprevious priority ranking is highest is not present or the ranking ofsuch a face region changes. In this way, the priority rankings of theother face regions change synchronously with the change of the priorityranking of the face region having the highest priority ranking. Hence,appropriate priority rankings in consideration of the state change ofthe main subject can be determined while suppressing frequent switchingin the priority rankings.

Furthermore, since the priority ranking of each subject can beappropriately determined, it is possible to not only determine the mainsubject, but also determine the priority ranking suitable for eachpurpose, such as AF, AE, and display, among which a priority subject maybe different.

Third Embodiment

The following describes a third embodiment of the present invention.When compared with the processing in the first and second embodimentsdescribed above, the third embodiment differs further in the timing inwhich the previous priority ranking result is not taken intoconsideration in the priority determination unit 310. That is, in thethird embodiment, the previous priority ranking result is not taken intoconsideration in the timing when an amount of change between thecurrently processed image and the image used when determining theprevious priority ranking is found to be not smaller than apredetermined amount. In the third embodiment, whether or not thecurrently processed image has a substantial change from the image usedwhen determining the previous priority ranking is detected.

FIG. 10 is a flowchart showing an example of processing for determiningpriority rankings of face regions according to the third embodiment. Forexample, the processes in this flowchart are executed by the controlunit 320 controlling the face detection unit 309, the prioritydetermination unit 310, and the priority ranking determination unit 311according to a program, as with the flowchart of FIG. 4 described above.

The processes of steps S1101 to S1104 are the same as the processes ofsteps S402 to S405 in FIG. 4 described in the first embodiment. Indetail, in step S1101, the face detection unit 309 performs a facedetection process on the image data of one frame to detect face regionsin the image, and obtains information on the detected face regions. Instep S1102, which follows, the control unit 320 determines whether ornot a priority has been computed for every face region detected in theimage. When the control unit 320 determines that a priority has beencomputed for every detected face region, the processing advances to stepS1108. In step S1108, the priority ranking determination unit 311determines a priority ranking of each face region detected in the imagebased on the computed priority, as in step S410 in FIG. 4.

When the control unit 320 determines in step S1102 that a priority hasnot been computed for every detected face region, on the other hand, theprocessing advances to step S1103 where the priority determination unit310 computes a position weight of a face region as in step S404 in FIG.4. In step S1104, which follows, the priority determination unit 310computes a size weight of the face region as in step S405 in FIG. 4.

The processing then advances to step S1105 where the prioritydetermination unit 310 determines whether or not there is a substantialchange in the captured image. For example, the change in the capturedimage can be detected based on the number of faces detected by the facedetection unit 309 in the image. In other words, when the number offaces detected in the image changes, the priority determination unit 310determines that there is a substantial change in the captured image.

The present invention is not limited to this. The change in the capturedimage may instead be detected based on a change in image captureconditions, such as a change of zoom information in the image capturingapparatus 300 or a change in an amount of blurring according to the gyrosensor. For instance, the zoom information can be obtained from theimage capture control unit 305, and the change in the amount of blurringcan be detected directly from the output of the gyro sensor or obtainedfrom the image processing unit 306. In the case where the valueindicating the image capture conditions changes at least by apredetermined amount, the captured image is determined to have asubstantial change.

When the priority determination unit 310 determines in step S1105 thatthe captured image has a substantial change, the processing advances tostep S1107. A process of step S1107 will be described later.

When the priority determination unit 310 determines in step S1105 thatthe captured image has no substantial change, on the other hand, theprocessing advances to step S1106. In step S1106, the prioritydetermination unit 310 obtains a weight of a previous priority ranking,that is, the most recent priority ranking, for the face region as instep S407 in FIG. 4. The processing then advances to step S1107 wherethe priority determination unit 310 computes a priority of the faceregion as in step S409 in FIG. 4. In detail, in step S1107, the prioritydetermination unit 310 computes the priority of the face region usingthe above expression (1), based on the position weight, the size weight,and the previous priority ranking weight determined respectively insteps S1103, S1104, and S1106 described above.

After the priority of the face region is computed, the processingreturns to step S1102. As mentioned earlier, when the control unit 320determines in step S1102 that a priority has been computed for everydetected face region, the processing advances to step S1108. In stepS1108, the priority ranking determination unit 311 determines thepriority ranking of each face region detected in the image in decreasingorder of priority based on the priority computed in step S1107, as instep S410 in FIG. 4. The determined priority ranking is held in thepriority ranking determination unit 311, and also associated with faceregion information and supplied to the image capture control unit 305.

After the priority ranking of each face region is determined by thepriority ranking determination unit 311 in step S1108, the processingreturns to step S1101. Note that the loop processing of steps S1101 toS1108 is repeated at a predetermined rate, such as a period of one frameor a period of a plurality of frames, or according to a time periodrequired for the face detection by the face detection unit 309.

The processing according to the flowchart of FIG. 10 mentioned above isdescribed in more detail below, with reference to FIG. 11. In an exampleshown in FIG. 11, four face regions 220 to 223 are each detected inframes #1 to #4, with there being a change in captured image betweenframes #2 and #3 as a result of driving the zooming mechanism.

In FIG. 11, in the first frame #1, there is no previous priority rankingdetermination result, so that the priority rankings are determinedaccording to the position weights and the size weights. In the exampleof FIG. 11, the face region 223 positioned nearest the image center isgiven the highest priority ranking. Meanwhile, the face region 220 isfarthest from the image center among the face regions 220 to 223, and sois given the lowest priority ranking.

In the next frame #2, the priority rankings are determined while takingthe previous priority rankings into consideration, for face regions thatare found to correspond to the face regions detected in frame #1. Thisbeing so, when there is no substantial change in the states of the faceregions in the image, a change in the priority rankings is suppressed.

The zooming mechanism is driven between frames #2 and #3, as a result ofwhich frame #3 has at least a predetermined amount of change of zoominformation when compared with frame #2. Accordingly, the prioritydetermination unit 310 determines in step S1105 that the captured imagehas a substantial change, so that the priority ranking of each faceregion is determined without taking the previous priority ranking resultinto consideration. Thus, the priority rankings of the face regions aredetermined according to the current states of the face detectionresults, as a result of which the state changes of the subjects arereflected more on the priority rankings.

In detail, in the example of FIG. 11, the face region 223 is given thehighest priority ranking in frame #3, because the face region 223 isnearest the image center and also largest in size among the face regions220 to 223. Meanwhile, the face region 222 whose priority ranking inframe #2 is second or third highest is given the lowest priorityranking, because the face region 222 is farthest from the image centeramong the face regions 220 to 223. The face region 220 whose priorityranking in frame #2 is lowest is given the second or third highestpriority ranking, because the face region 220 is nearer the image centerthan the face region 222.

The third embodiment describes the case where the priority ranking ofeach face region is determined without taking the previous priorityranking into consideration when there is a change in the captured image.However, this example is not a limit on the present invention. Forinstance, the priority ranking may be determined by changing how muchthe previous priority ranking is taken into consideration, depending onthe change in the captured image.

According to the third embodiment described above, frequent switching ofthe priority ranking of each face region is suppressed during the timeperiod when the previous priority ranking is taken into consideration.Moreover, by providing a timing in which the previous priority rankingis not taken into consideration or how much the previous priorityranking is taken into consideration is changed, a priority ranking thatreflects the state of the subject at a specific time is determined. Thatis, the previous priority ranking is not taken into consideration or howmuch the previous priority ranking is taken into consideration ischanged, in the case where the captured image is determined to have asubstantial change. In this way, switching in the priority rankings issuppressed when the change in the captured image is small, whereas thecurrent states of the subjects are given precedence when the change inthe captured image is large. Hence, appropriate priority rankings can bedetermined by promoting switching in the priority rankings according tothe change in the captured image, while suppressing frequent switchingin the priority rankings.

Furthermore, since the priority ranking of each subject can beappropriately determined, it is possible to not only determine the mainsubject, but also determine the priority ranking suitable for eachpurpose, such as AF, AE, and display, among which the priority subjectmay be different.

Each of the above embodiments describes the case where, when thepriority determination unit 310 computes the priority, the priorityranking previously determined by the priority ranking determination unit311 is not taken into consideration or how much the previous priorityranking is taken into consideration is reduced, according to thepredetermined condition. Here, during a predetermined time period afterthe previous priority ranking is not taken into consideration or howmuch the previous priority ranking is taken into consideration isreduced, the previous priority ranking result may be taken intoconsideration in the priority computation regardless of such apredetermined condition. This prevents frequent occurrences of thepriority computation process in which the previous priority ranking isnot taken into consideration or how much the previous priority rankingis taken into consideration is reduced, with it being possible tosuppress frequent switching in the priority rankings.

Each of the above embodiments describes the case where a face isdetected as a subject, but the present invention is not limited to thisexample. The present invention is applicable to any instance where asubject of a specific shape is detected in order to extract an intendedsubject such as a person, an animal, or a car.

Each of the above embodiments describes the case where the presentinvention is applied to an image capturing apparatus, but the presentinvention is not limited to this example. The present invention is alsoapplicable to a reproduction apparatus that reproduces a moving image.In the case of applying the present invention to such a reproductionapparatus, subject detection is performed on reproduction data forreproducing the moving image, and a priority ranking of each detectedsubject is determined. The determined priority ranking is associatedwith face region information and supplied to the image processing unit306. This enables the image processing unit 306 to perform imageprocessing on each subject according to the priority ranking.

Other Embodiments

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment(s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment(s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (e.g., computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2008-316275, filed on Dec. 11, 2008, which is hereby incorporated byreference herein its entirety.

1. An image processing apparatus that determines a priority ranking of asubject detected from an image, the image processing apparatuscomprising: a detection unit that detects one or more subjects from animage; a computation unit that computes a priority for determining apriority ranking, for each of the subjects detected by the detectionunit; and a determination unit that determines the priority ranking ofeach of the subjects detected by the detection unit, based on thepriority computed by the computation unit, wherein the computation unit:assigns a larger weight to a subject detected by the detection unit atleast one of when the subject has a larger size and/or when the subjectis positioned nearer a center of the image; corrects the assigned weightby a larger weight when a most recent priority ranking determined forthe subject by the determination unit in a previous image from which thesubject is detected is higher; computes a higher priority for thesubject when the corrected weight is larger; and, when a predeterminedcondition is satisfied, computes the priority with a reduced amount ofcorrection by the weight of the most recent priority ranking.
 2. Theimage processing apparatus according to claim 1, wherein, when thecomputation unit has performed the priority computation a predeterminednumber of times, in a next computation the computation unit computes thepriority with the reduced amount of correction by the weight of the mostrecent priority ranking, the predetermined number being not less than 2.3. The image processing apparatus according to claim 1, wherein thecomputation unit obtains an amount of change between the image and theprevious image used when determining the most recent priority rankingand, when the amount of change is determined to be not smaller than apredetermined amount, computes the priority with the reduced amount ofcorrection by the weight of the most recent priority ranking.
 4. Theimage processing apparatus according to claim 3, wherein the computationunit determines that the amount of change is not smaller than thepredetermined amount, when a subject whose most recent priority rankingis highest is not detected by the detection unit.
 5. The imageprocessing apparatus according to claim 3, wherein the computation unitdetermines that the amount of change is not smaller than thepredetermined amount, when the number of the subjects detected by thedetection unit is different from the number of subjects detected fromthe previous image used when determining the most recent priorityranking.
 6. The image processing apparatus according to claim 3,wherein, during a predetermined time period after computing the prioritywith the reduced amount of correction by the weight of the most recentpriority ranking, the computation unit computes the priority withoutreducing the amount of correction by the weight of the most recentpriority ranking, regardless of how large the amount of change is. 7.The image processing apparatus according to claim 1, wherein thedetection unit detects human faces as the subjects.
 8. An imageprocessing method for determining a priority ranking of a subjectdetected from an image, the image processing method comprising: adetection step of detecting one or more subjects from an image; acomputation step of computing a priority for determining a priorityranking, for each of the subjects detected in the detection step; and adetermination step of determining the priority ranking of each of thesubjects detected in the detection step, based on the priority computedin the computation step, wherein the computation step includes:assigning a larger weight to a subject detected in the detection step atleast one of when the subject has a larger size and/or when the subjectis positioned nearer a center of the image; correcting the assignedweight by a larger weight when a most recent priority ranking determinedfor the subject in the determination step in a previous image from whichthe subject is detected is higher; computing a higher priority for thesubject when the corrected weight is larger; and, when a predeterminedcondition is satisfied, computing the priority with a reduced amount ofcorrection by the weight of the most recent priority ranking.
 9. Animage capturing apparatus comprising: an image capture unit thatcaptures images based on light incident via an imaging optical system,and that sequentially outputs the images; and the image processingapparatus according to claim 3 that processes the images output from theimage capture unit.
 10. The image capturing apparatus according to claim9, further comprising a zoom drive unit that drives a zooming mechanismincluded in the imaging optical system, wherein the computation unitdetermines that the amount of change is not smaller than thepredetermined amount when the zooming mechanism is driven by the zoomdrive unit between times of capturing the images for which the amount ofchange is obtained.
 11. The image capturing apparatus according to claim9, further comprising a sensor unit that detects a movement of the imagecapturing apparatus, wherein the computation unit determines that theamount of change is not smaller than the predetermined amount when amovement not smaller than a predetermined amount is detected by thesensor unit between times of capturing the images for which the amountof change is obtained.
 12. The image capturing apparatus according toclaim 9, wherein the image capture unit includes a focusing control unitthat performs focusing control of the imaging optical system based onthe priority ranking determined by the determination unit.
 13. The imagecapturing apparatus according to claim 9, wherein the image capture unitincludes an exposure control unit that performs exposure control of theimaging optical system based on the priority ranking determined by thedetermination unit.
 14. The image capturing apparatus according to claim9, further comprising a display unit that displays each of the imagesoutput from the image capture unit, wherein the display unit displaysthe image in which subjects detected from the image are associated withpriority rankings determined for the subjects, based on the priorityranking determined by the determination unit.